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ABSTRACT 

The thesis investigates the characteristics of effective teaching 
as perceived by students at the Naval Postgraduate School. Principal 
components factor analysis is used to extract the characteristics from 
observations on an 86 variable questionnaire form designed by Hildebrand 
and Wilson. The characteristics are then used as a basis for a short form 
questionnaire. Cluster and discriminant analysis are used to find teaching 
patterns or styles based on seven characteristics. Ranking schemes for 


evaluated instructors are also discussed. 
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I. INTRODUCTION 


А, MOTIVATION 

Within the last decade, an increasing emphasis on basing evaluation 
of teaching and of teachers on performance in the classroom environment 
has taken place at many college level academic institutions throughout 
this country. Prior to this time, the evaluation of teaching performance 
of university professors took a second place to other activities that were 
more readily observable, such as research work published, committee 
work, or performance of administrative tasks. The demands of adminis- 
tration positions often preclude even token visits to classroom sessions. 

Academic ME anos are, for the most part, former teachers 
themselves and thus probably tend to evaluate their colleagues as other 
teachers do. A survey conducted on the Davis Campus of the University 
of California provides evidence that teachers do not primarily judge each 
other on teaching ability in the en. In this survey, 162 teachers 
chose what they considered to be the best and worst of their colleagues 


while also answering some informational questions concerning themselves. 


Коза. J. W., "Policies « Practices in Faculty Evaluation," The 
Educational Record, v. 42, p. 194-211, 1961. 


M ton Hildebrand and R. C. Wilson, Effective University Teaching 
and Its Evaluation, report to the faculty sponsored by the Academic 
Senate, April 1970, p. 20. 





Fifty-one percent of the respondents had not observed a classroom per- 
formance of those chosen as best, while seventy-five percent had not 
done so for those chosen as worst. Admittedly, these judgements were 
relative to the respondent's own concept of what constituted a good or 
bad teacher, but it is surprising that a judgement can be made without 
some direct input from the classroom environment, the arena for a 
teacher's primary function. The results of this survey, coupled with the 
acknowledged burdens of academic administration, facilitate an under- 
standing of how classroom observation has come to contribute so little 
to the administrators' evaluation of his faculty. 

One form of the recent widespread effort to correct this deficiency 
has been the utilization of student opinion collected through questionn- 
aires or course critique sheets. Some cf this effort had its inception witn 
student body organizations motivated by a general student desire to have 
a say concerning the quality of the teaching he receives while striving 
to attain his academic goals. At the same time, many academic adminis- 
‘trations and faculty organizations have recognized that input from the 
classroom should play a greater role in teaching evaluation and that the 


student is qualified to provide this input. 


Slaten: R.S., Githens, W. H., and Senger, J. D., Factors 
Leading to Satisfaction and Dissatisfaction With Teachers, paper 


presented at the Western Psychological Association Convention, 
26 April 1972. 





As a result of such effort, colleges and universities have developed 
numerous student evaluation forms, largely with the same objectives, 
but each different enough to have questionable applicability at any other 
institution. Because there are unique factors associated with a particular 
college or university, such as general make up of the student body, type 
of funding, overall faculty profile, and the type education offered, each 
such institution should attempt to develop its own means of tapping its 
largest, most up to date, and readily available data source on classroom 


performance of its teachers, the students. 


b. BACKGROUND 

In September 1971, academic administrators and faculty at the 
Naval Postgraduate School initiated activity intended to determine what 
constituted effective teaching at that institution and to develop a means 
to measure it. Such measurements could then be used to assist the 
teacher in achieving his goals in his profession and could be used to 
aid the administration in its teacher evaluation task. The activities at 
the Naval Postgraduate School took several different forms, two of which 
are NE to this thesis. 

The first of these activities was the establishment of a committee 
within the Operations Research/Administrative Sciences Department 
tasked with determining the characteristics of effective teaching. Іп 
March 1972, after preliminary research, a report was published which 


identified the need for a thorough study to accomplish the following: 
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T Identify the ways that students look at BU SUUCLOFS. 


2. Number and characterize the significant dimensions of the 
teaching-learning process as viewed by our students. 


За Identify patterns or stylistic differences among teachers. 

4. Identify and quantify the importance of exogenous variables 
(еа subject matter, class size, core course, service 
се ete.. 

The other form was the establishment of a Faculty Council Committee 
tasked with finding improved measures of teaching performance. Of the 
approaches suggested, one of those adopted was the employment of the 
recently published methods developed by Milton Hildebrand and Robert 
C. Wilson at the University of California ‚Davis Campus E In December 
1971 and March 1972, an instructor evaluation form developed by them 
was administered to students from various curricula at the Naval Post- 
graduate School. It was hoped that their methodology applied to these 
data would yield results similar to theirs and lead to an efficient means 


of obtaining student opinion of teaching and teachers. (See Appendix A 


for a summary of data collected.) 


SI Postgraduate School, On the Quantification of Teacher 


Performance Using Student Opinion, by R. R. Read and H. J. Zwieg, 
p. 44, March 1972. 


MEM Hildebrand, R. C. Wilson, and E. R. Dienst, Evaluating 
University Teaching, University of California Press, 1971. 
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e DISCUSSION OF PREVIOUS RESEARCH 

Because the methods of Hildebrand and Wilson were used to generate 
the initial data and their results provide a basis of comparison for appli- 
cation of such methods, their work warrants description to the extent 
that it relates to this project. As part of an objective to characterize 
effective teaching performance and provide a basis for teaching evaluation, 
they developed a method to extract the components of effective teaching 
from student opinion questionnaires. They first developed what was 
termed a long form by reducing an initial list of 236 descriptions of 
various aspects of teaching to 85 items which were judged by students to 
be most discriminating between previously identified best and worst 
teachers. When this list had been put into questionnaire form, 119 
students at the University of California, Davis Campus indicated appli- 
cability to instructors to whom they had been exposed by simple yes, no, 
or no opinion responses. These data were further reduced by application 
of principal components factor analysis with varimax rotation and yielded 
the following components of effective teaching: 
Synthetic/Analytic Approach 
Organization/Clarity 
Instructor-Group Interaction 


Instructor-Individual Interaction 
Dynamism/Enthusiasm 


сл 4» (9 ГӘ нә 


The above descriptions of the resulting components are subjective to the 
extent that they are based on an interpretation of those original variables 
most highly correlated with each principal component of the solution. 


These same variables, put together in a verbal description of the 
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components, are the basis for a short form questionnaire. When both 
the long and short forms were administered simultaneously, the short 
form variables were found to have high correlations with the original 
variables on the long form. Table I shows the long form, annotated to 
reflect those variables which contributed significantly to the principal 
components solution, along with these variables' correlations witn the 
components. These correlations are called factor loadinas. Table II 


shows the short form variables describing the principal components. 


DR APPROACH 

This research began in October 1972 with the acquisition of the 
raw data gathered under the sponsorship of the Faculty Council committee. 
The overall objective was to use these data and any other that might be 
obtained to satisfy the objectives of both the Faculty Council committee 
and the Operations Research/Administrative Sciences committee. To this 
end, the following basic approach was planned: 

ЈЕ. Employ the methodologies of factor analysis on the initial 
data to determine if lts existed which were sharper than Hildebrand 
and Wilson's . 

2 Employ the chosen factor analysis method to determine the 
components of effective teaching from the data with attention to curricula 
as an exogenous variable. 

3: Use the components found to develop a short form question- 


naire and determine its applicability and validity. 
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4. Employ cluster analysis to determine patterns or styles of 
teaching based on the components of effective teaching previously 
determined. 

SA Investigate the dimensionality of student perception of 
teaching and their evaluations of teachers. 

65 Respond to other questions that present themselves during 


mae course ог Ше research. 
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PRINCIPAL 
COMPONENT 


1 
1 
1 


~ 


сл п сл сл wo 


TABLE I 


LIST OF CHARACTERISTICS 


LOADING 


. 66 
250 
so 


Us 
62 


E 
290 
‚47 


80 
.74 
725 
.64 
SIDES 
98 


COURSE CONTENT AND PRESENTATION: 


pal 
» 


10: 
rh. 
ШЕ 
13. 


14. 


Loc 
16. 
17. 
er 
197 


ZUR 
21% 
2 25; 
23% 
24. 
257 
256 
Zur 
28: 


296 
30. 


Contrasts implications of various theories 
Presents origins of ideas and concepts 
Presents facts and concepts from related 
fields 

Talks about research he has done himself 
Emphasizes ways of solving problems 
rather than solutions 

Discusses practical applications 
Explains his actions, decisions, and 
selection of topics 

Seems well read beyond the subject he 
teaches 

Is an excellent public speaker 

Speaks clearly 

Explains clearly 

Gives lectures that are easy to outline 
Reads his lectures or stays close to his 
notes 

Assigns text as background, but lectures 
include other topics 

Makes difficult topics easy to understand 
Summarizes major points 

States objectives for each class session 
Identifies what he considers important 
Shows interest and concern in quality 

of his teaching 

Gives examinations requiring creative, 
original thinking 

Gives examinations having instructional 
value 

Gives examinations requiring chiefly 
recall of facts 

Gives interesting and stimulating 
assignments 

Stresses the aesthetic and emotional 
value of the subject 

Is a dynamic and energetic person 
Seems to enjoy teaching 

Is enthusiastic about his subject 

seems to have self-confidence 

Varies the speed and tone of his voice 
Has a sense of humor 
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PRINCIPAL 
COMPONENT 


2 


MALLET Cont'd) 
OTOP CHARACTERISTICS 


LOADING 


.61 


E) 


265 


.64 


5122 
198 


257 


.43 


se 
32 
33. 
34. 
35% 


36. 
37. 


38. 


39% 


40. 


41. 


42, 


43. 
44. 


45. 


46. 


47. 


48. 


49. 


50. 
Sr 


SM 


OE 


RELATIONS WITH STUDENTS: 


Is careful and precise in answering 
questions 

Explains his own criticisms 

Encourages class discussion 

Invites students to share their knowledge 
and experiences 

Clarifies thinking by identifying reasons 
for questions 

Invites criticism of his own ideas 
Knows if the class is understanding him 
or not 

Knows when students are bored or 
confused 

Has students apply concepts to demon- 
strate understanding 

Keeps well informed about progress of 
class 

Anticipates difficulties and prepares 
students beforehand 

Has definite plan, yet uses material 
introduced by students 

Provides time fcr discussion and questions 
Is sensitive to student's desire to ask 
a question 

Encourages students to speak out in 
lecture or discussion 

Quickly grasps what a student is asking 
or telling him 

Restates questions or comments to 
clarify for entire class 

Asks others to comment on one student's 
contribution 

Compliments students for raising good 
points 

Answers questions fully 

Determines if one student's problem is 
common to others 

Reminds students to see him if having 

са д ЕУ 

Informs students of coming campus 
events related to course 
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PRINCIPAL 
COMPONENT 


> 


Au 


TABLE I (Cont'd) 
LIST OF CHARACTERISTICS 


LOADING 


.74 
.69 


. 68 
. 64 


TA 


. 64 


295 


.46 


N59 


и | 
2196 
‚60 


RELATIONS WITH STUDENTS (Cont'd) 


54. 
DN 
obe 
S 
98: 
S95 


60. 


b 


Ба. 


63. 


64. 
бо. 


66. 


OF. 
68. 


DIR 
70. 


7 


or 


Дон 
74. 
73. 
76. 
vie 


Encourages students to express feeling 
and opinions 

Relates class topics to students' lives 
and experiences 

Has a genuine interest in students 
Relates to students as individuals 
Recognizes and greets students out of 
class 

Is valued for advice not directly related 
to the course 

Treats students as his equals 


OTHER CHARACTERISTICS: 


Discusses points of view other than 

his own 

Discusses recent developments in the 
field 

Gives references for the more interesting 
and involved points 

Emphasizes conceptual understanding 
Disagrees with some ideas in textbook 
and other readings 

Stresses rational and intellectual aspects 
of the subject 

Stresses general concepts and ideas 
Seems to have a serious commitment 

to his field 

Is well prepared 

Gives examinations stressing conceptual 
understanding 

Gives examinations requiring synthesis 
of various parts of course 

Gives examinations permitting students 
to show understanding 

Is friendly toward students 

Is accessible to students out of class 
Respects students as persons 

Is always courteous to students 

Gives personal help to students having 
difficulty with course 
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TABLE I (Cont'd) 
LIST OP CHARACTERISTICS 


PRINCIPAL 


COMPONENT LOADING OTHER CHARACTERISTICS (Cont'd) 


5 76 le. 
* 797 


Has an interesting style of presentation 
Was constrained by having to meet a 
rigorous course outline 


SUMMARY OF IMPRESSIONS OF INSTRUCTOR'S 


COURSES: 

80. Have developed increased appreciation 
for the subject 

81. Have learned new ways to evaluate 
problems 

82. Have worked harder than in most other 
courses 

83. Know how to find more information on 
the subject 

84. Have studied a topic from the course 
on own initiative 

85. Plan to take more courses on the subject 

86. Have gained self-knowledge 


* 
This variable was not on Hildebrand and Wilson's original list 
but was added by the Faculty Committee 
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TABLE II 


SHORT FORM VARIABLES DEVELOPED BY HILDEBRAND AND WILSON FROM 
THE PRINCIPAL COMPONENTS SOLUTION OF THEIR LONG FORM STUDENT 
OPINION QUESTIONNAIRE 


b 


(Synthetic/Analytic (М сасһ)? Has command oí the subject, presents 
material in an analytic way, contrasts various points of view, dis- 
cusses current developments, and relates topics to other areas of 
knowledge. 

(Organization/Clarity) Makes himself clear, states objectives, 
summarizes major points, presents material in an organized manner, 
and provides emphasis. 

(Instructor-Group Interaction) Is sensitive to the response of the 
class, encourages student participation, and welcomes questions 
and discussion. 

(Instructor-Individual Interaction) Is available and friendly towards 
students, is interested in students as individuals, is respected as 

a person, and is valued for advice not directly related to the course. 
(Dynamism/Enthusiasm) Enjoys teaching, is enthusiastic about 


his subject, makes the course exciting, and has self-confidence. 


6 | 
Parenthetical remarks were not included on the short form 


questionnaire but have been included here tor clarity. 
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Il. THEORY OF VARIABLES 


A. FACTOR ANALYSIS MODEL 

Factor analysis, in the most simplistic definition, is a mathematical 
means of finding a more parsimonious representation of a data set. the 
variables of which are too numerous and too complex in their inter- 
relationships to be efficiently analyzed by the more basic statistical 
methodologies. It seeks a more regular and orderly representation through 
a reduction in the number of variables, while retaining a significant pro- 
portion of the information contained in the original data's variance-co- 
variance structure. ) There are two basic types of factor analysis, principal 
components and classical. The former is generally used when extraction 
of a maximum of the variance is desired with the reduction of a large body 
of data, while the latter is used when a maximal reproduction of the cor- 
relations is а In addition, there are a number of methods of 
solution improvement, called rotations, which have been developed to 


render the factor structure more easily interpretable к; 


aan LN соавт Factor Analysis, p. 15, The University 
of Chicago Press, 1967. 


on. D. F., Multivariate Statistical Methods, p. 277-286, 
McGraw-Hill, 1967. 
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Principal component factor analysis with Kaiser's varimax rotation 
was chosen for this research effort primarily because it was used by 
Hildebrand and Wilson in their work and it was anticipated that a com-- 
parison of results could be made. 

The principal components method is based on the modei that the 
original variables, the characteristics listed in the long form questionnaire 
in this case, can be expressed in a linear combination of hidden or latent 
variables called the principal components. Algebraically: 

E x + E +- - - - - - МЕ 
мћеге 2. is the standardized representation of the original variable x; ; 

j=1, 2, -------- , n variables, 

i=l], 2, -------- , N observations. 

The E155 e new uncorrelated, mutually orthogonal components repre- 
senting TA Trent variables and the as are the correlations, called 
factor loadings, of the original variables with the components n 

In a geometric interpretation, the original data in n-space is pro- 
jected onto mutually orthogonal axes. These axes are then rotated so 
that the first axis is aligned in the direction of maximum variance of the 


original data and each succeeding axis is aligned in the direction of next 


largest variance maintaining orthogonalty during the successive rotation. 


9 | i 
If the model is for the non-standardized representation of the 
original variables, the NS are not considered as correlations. 
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In the model, F, represents the dimension of greatest variability and for 


1 
the data in question, is the component of teaching most discriminated 
upon by the respondents. The а 1! j=1, 2, -------- , n, represents the 
contribution of the original variable j to the first principal component, Pie 
Geometrically, it may be interpreted as a direction cosine. As a simple 
geometric example, consider Figure 1. In this figure, three original 
variables are represented by the solid axes. The variables are height, 
weight, and hair length of a population of generally short haired people. 
The data points plotted show considerable variability in the first two 
dimensions and little in the third. Now, rotating the axes and aligning 
one with the direction of greatest variance (represented by the broken 
lines), shows virtually all the variability as being in one dimension. 
Because the example is known to have little variability with regard to hair 
length, the other variables are the ones which contribute to the make up 
of the new variable which might be called "size." The contributions of 
the original variables to the new variables are measured by the direction 
cosines of the angles between the old axes and the new. This example 
illustrates essentially what happens in principal components analysis 
with the exceptions that: there are generally many more dimensions to 
start with, and a very small percentage of the new dimensions contain a 
large percentage of the original variance. 

A more rigorous explanation is given by Anderson, presented here 


— 


without proof: 
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FIGURE 1. GEOMETRIC EXAMPLE 
PRINCIPAL COMPONENTS ANALYSIS 





HEIGHT 


23 





Theorem: 
Let the n- component random vector Z have E(Z) = 0 and VAYA) = >, 
Then there exists an orthogonal linear transformation U = В 7 such that 
the covariance mye Oiled E(U*U) = Á and 
1 О 
Ша Ao 
O > X 
n 
where А 1 < THE сен < A , are the roots of 
EAN = 0. The rth column of B, в) satisfies (> = Al] В - 0. 
(г) 


The rth component of U, U =B Zand has maximum variance of all 


10 


linear combinations uncorrelated with U., U, ----, U] 


1 2 

For this model, the B matrix is the eigenvector matrix of the original 
data's covariance structure and its entries, the I are the factor 
loadings. The A ‚ 5 are the proportion of the total variance associated 
with component i. The sum of the M i=1,----- ,n, equals the total 
variance of the principal components and is alsc equal to the total variance 
of the original data. The number ya is also the eigenvalue of com- 
ponent DU E ----- pony. 


There are, according to the above Theorem, as many new variables 


after rotation and projection as there were originally. Parsimony is 


10 | 
Anderson, T. W., An Introduction to Multivariate Statistical 


nal Sip 0276, Jonn Wiley and Sons, inc. , 1958. 
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realized when a significant percentage of the original variance is repre- 
sented by the first few components. These are retained and the remaining 


components can be ignored. 


Бе PRINCIPAL COMPONENT INTERPRETATION 

There may be a problem in deciding just how many of the principal 
components to retain when arriving at a final solution when the number 
of original variables is large. When the number is not large, there are 

| ; пет ДЕН 

tests which may be applied which indicate the proper cutoff. These 
tests will also give a result for the case of a large number of original 
variables, but the result may indicate that many more components should 
be retained than are necessary or desired. For example, an exchange of 
86 variables for 55 new variables will most likely not be a useful data 
reduction. What is commonly done is to base the number to be retained 
on prior knowledge of the structure of the data or to retain only those 
components that can be interpreted from the factor loadings. Morrison 
states: 

"It has been the author's experience that if that proportion (ап 

arbitrarily large percentage of the total variance) cannot be 

explained by the first four or five components, it is usually 

fruitless to persist in extracting vectors (components), for even 

if thelater characteristic roots (eigenvalues) are sufficiently 


distinct to allow easy computation of the components, the in- 
terpretation of the components may be difficult if not impossible." 


ON АСИ р. 219-221; 


ол, DEIS A 228. 
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Interpretation of a principal component is Pompo ned by sub- 
jective evaluation of the interaction of the factor loadings, the е Я 
in the column of the loading matrix associated with the component. Recall 
that these loadings are the correlations of the original variables with the 
principal components. By considering those variables with the highest 
factor loadings it is possible to ascribe an overall impression of what 
the component description is. For example, consider Table III which 
shows the first column of the factor loading matrix from a principal com- 
ponents solution on the initial data. The highest loadings are underlined 
and ordinally ranked. In Table IV, these loadings, the variables with 
which they are associated, and the variable description are listed in de- 
creasing order of contribution to the component. The original variables 
11, 31, 37, 50, and 15 seem to imply a characteristic oí clarity while 
12, 16, 69, and 17 seem to imply organization. A possible subjective 
description of the latent variable, the principal component, would be 
Clarity and Organization. 

The interpretation of each successive principal component is 
accomplished in the same manner. The order of interpretation is that of 
decreasing contribution to the total variance or the decreasing eigenvalues. 
However, as the eigenvalues associated with the components decrease, 
so too do the magnitudes of the higher factor loadings, the tendency 
being for each column to show a greater number of the original variables 


with loadings that are considered to be high for the column but low 
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TABLE III 


БОА ОЕ ТИНЕ FIRST COLUMN OF THE MATRIX OF COEFFICIENTS ОР 
ice RI NGIFAL COMPONENTS FACTOR SOLUTION TO BE USED TO INTERPRET 
ШИЕ DESCRIPTION OF THE FIRST FACTOR 


VARIABLE LOADING VARIABLE LOADING 
Т SO 44 . 17808 
2 „24275 45 15283 
3 -40222 46 .43623 
4 ПӘЛЕ? 47 . 18707 
3 ‚27606 48 USO 
6 77 49 209479 
7 . 40413 50 . 60175 5 
8 298579 51 AMS 
9 Tool 52 .08014 

10 are Tol 520) -гошиса 
11 .66136 | 54 410357 
2 ‚61633 3 Sis wl Oren 
13 ӘЛІ > 56 „297556 
14 200276 Sm ЕЛБА 
ща 2904920 7 58 09250 
16 оао 8 99 .34866 
17 ТО 60 ‚24600 
18 .46128 61 .09676 
19 АЛИА 62 · 16384 
20 2220/08 63 20075 
2 · 34127 64 20008 
22 -. 00843 65 .03449 
23 250951 66 ASS 
24 . 00475 67 . 16664 
25 120287 68 .35832 
26 128895 69 -90099 2 
27 222526 70 сео 
28 ‚ 14762 71 ‚26968 
29 207290 Да 529095 
30 - 12059 73 -07/791 
21 . 63895 2 74 .06014 
27 . 19004 Zo „Ra | 
33 03289 76 -15093 
34 .07870 77 . 17045 
35 „24359 78 ӘЗІЗ Y 
36 . 14032 79 07016 
37 . 60848 4 80 20990 
38 . 46292 81 „24285 
39 . 24362 82 217930 
40 . 56442 6 83 „13587 
41 . 46630 84 =„03767 
42 221059 85 . 10786 
43 1599 86 · 08458 
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TABLE IV 


LIST OF VARIABLE DESCRIPTIONS IN ORDER CF DECREASING LOADINGS 


FOR EXAMPLE IN TABLE III 


LOADING 


.66136 
163895 
. 61633 
. 60848 
#60175 


.56442 


.54326 
(59656 
250099 


.48285 


VARIABLE + 


11 


Sl 


12 


37 


50 


40 


Шъ 


16 


69 


17 


DESCRIPTION 
Explains clearly 
Is careful and precise in answering questions 
Gives lectures that are easy to outline 
Knows if class is understanding him 
Answers questions fully 


Keeps well informed about the progress of 
the class 


Makes difficult topics easy to understand 
Summarizes major points 
Is well prepared 


States objectives for each class session 
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relative to the loadings on the previously determined components. This 
makes interpretation more difficult than for earlier components. At some 
point, the value of the associated eigenvalue will be less 1.0. This 
implies that the component contributes less to the total variance than 
did any one of the original variables. Those components with associated 
eigenvalues greater or equal to 1.0 and with factor loadings that render 
the component interpretable are the ones to be retained. 

With respect to the data of this research effort, the retained com- 
ponents constitute a model of the components of effective teaching as 
perceived by the students, and their number represents the dimensionality 
of such perceptions. It is important to note that the components describe 
what the students discriminate on among teachers js a result of the 
dependence of the method on the variance-covariance structure of the 
original data. It is quite possible that other components of effective 
teaching may be considered important by the respondents but may fail 
to show up in the principal components solution. For example, should 
all of the instructors score high on those variables that contribute to a 
component that might be interpreted as Evaluation Technique, the con- 
tribution of this component to the total variance would be small compared 
to that of components with a full range of high and low marks. In this 


case, the component might not show up in the final solution. 
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om FACTOR SCORES 

While meaningful results are embodied in the principal component 
solution as it stands, the descriptions and dimensionality can be used 
to gain further useful information. The vector of responses on the original 
variables can be transformed into a smaller vector of responses on the 
new variables which in theory retain a significant amount of the original 
information. It then becomes tenable to apply other methods of statistical 
analysis, which would be intractable otherwise, to the data in its reduced 
dimensioned form. For example, the results of a cluster analysis applied 
to a data set of 86 variables would probably be difficult to interpret, and 
require unreasonable computing power. However, the same methodology 
applied to the much smaller number of variables resulting from the prin- 
cipal components solution would usually have more easiiy interpreted 
results. A factor scoring transformation matrix is used to accomplish 
this reduction of the data's dimension. 

The factor scoring transformation matrix is computed by multipli- 
cation of the matrix of component ccrrelations, the matrix of factor 
loadings, and the variance-covariance matrix of the original data. The 
resulting matrix is then applied to each vector oí responses from tne 
original data. This yields a vector which has a one to one correspondence 
with the characteristics or effective teaching as determined by the 
principal components solution and may be thought of as the responses 
that the student might have given had ne been asked to evaluate t 


ticular instructor directly on the new variables. 
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Given a principal components solution of p components, the individ- 


ual entries in the factor scoring transformation matrix of size (pxn) are 


computed by the BMDX72 Factor Analysis Program as follows: > 


ве = 2 Gia ab Бе 
a mu m 


where Да is the correlation of component j with component a 


Uat is the factor loading of original variable b on component a 


Soc is the covariance of original variable b with original 


variable c 


HEIN M LLL —_ mo 
Је === d 
UM uc /П 
ЖОККО == ‚р 
n = number of original variables 


p = number of retained components 
The vector of p factor scores for observation i is then computed by multi- 
plying the vector of n original data scores by the above matrix. The 


computation of the individual scores of the vector of factor scores is as 


follows: 


250-3 Dou vsUniVersitv Of California Publications in Automatic 
Computations, No. 3, BMD Biomedical Computer Programs, X-series 
Supplement, p. 90-103, University of California Press, Berkeley and 
Los Angeles, 1969. 
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а » se 
) С ) 


where Zi is the standardized score on original variable c for 


observation i 


N = Total number of observations 
Each li is a representation of the information contained in the original 
data expressed in terms of the component of effective teaching j for each 
observation i. Some information is lost in the transformation but this 
loss is offset by the more parsimonious representation of the students' 


Opinions or perceptions. 


B. ТНЕ SHORT FORM VARIABLES 

The ultimate worth of the determination of the components of 
effective teaching is their employment in the development of a short form 
questionnaire that will not overly tax the respondent but will still produce 
the information desired. The approach used by Hildebrand and Wilson 
was to combine the descriptions of the original variables used to interpret 
each component to form a single composite description of that component 
and use it as a short form variable. Responses were to be made on a 
psychological scale of 1 to 7 to indicate degree of applicability to the 
evaluated instructor. To help illustrate how this is done, the original 
variable descriptions listed in Table IV, which led to an interpretation 
ofa а called Organization and Clarity, are reproduced below 


in order of decreasing factor loadings. 
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VARIABLE NUMBER DESCRIPTION 


11 Explains clearly 

S Is careful and precise in answering questions 

12 Gives lectures that are easy to outline 

37 Knows if class is understanding him 

50 Answers questions fully 

40 Keeps well informed about the progress of 
the class 

15 Makes difficult topics easy to understand 

16 Summarizes major points 

69 Is well prepared 

17 States objectives for each class session 


These descriptions might be reworded and put together as follows: 

He makes himself clear, makes his objectives known, 

summarizes major points, is well prepared, presents 

material in an organized manner, and is aware of class 

progress and understanding. 
The subjective interpretation (Organization and Clarity) does not appear 
in the composite description so the respondent remains focused on the 
combination of original variables and is not prompted to respond on the 
interpretation alone. Short form variables are similarly constructed for 
each of the components resulting from a given principal component 


solution so that the short form questionnaire will have a variable for 


each component. 


fi. LADDER SCORES ' 
In order to compare this study with that of Hildebrand and Wilson 
and to give direction to the components discovered, it was necessary 


to compile lists of best and worst instructors. It had been shown in 
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previous research that a ladder technique can serve this purpose. 3 The 
technique provides additional data on the respondents ranking of each 
instructor relative to the other instructors he evaluated. For these reasons, 
the ladder ranking of instructors was required of each respondent for all 
data sets collected for this study. 

For completeness, a description of the ladder technique follows. 
Figure 2 shows a fifteen step ladder which was used to represent the 
students one dimensional ranking of teachers. To use the ladder, the 
student was asked to think of the very best and very worst teachers he 
had ever been exposed to in his entire academic experience and place 
them at the top and bottom of the ladder, respectively. He was then to 
rank all teachers he had been exposed to from an eligible teacher list 
provided. Ties were permitted. He also was to indicate a minimum 
acceptable level of teaching performance on this ladder. 

The rankings were converted to range scaled scores (0.0 to 1.0) 
for each student and then averaged for each instructor. This provided a 
simple representation of the ranking by the students, as a group, for all 
instructors evaluated. This ranking might then be used for comparison 
with ranking schemes constructed from the questionnaire data in original 


or transformed form. 


4 
| Read and Zwieg, op. cit., p. 5-8. 
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FIGURE 2. LADDER SCALE OF OVERALL 
TEACHING EFFECTIVENESS EVALUATION FORM 


ABSOLUTE BEST 


ЛЕ) 
14 
13 


|2 


ABSOLUTE WORST 


A tio nn a m ee + 





35 





III. ANALYTICAL RESULTS 


A. PRINCIPAL COMPONENTS ANALYSIS 

Determination of what constitutes effective teaching is the most 
important objective of this research effort. It was anticipated that the 
methods of Hildebrand and Wilson would be used to accomplish this. It 
was first necessary to ensure that their method could not be improved 
upon. The first step was then that of subjecting the initial data, col- 
lected under the sponsorship of the Faculty Council Committee, to both 
types of factor analysis with all possible rotation options. This was 
done with the December 1971 data and there was no appreciable improve- 
ment of any one solution over any other. Therefore all subsequent factor 
analysis was done using the principal components method. As a by- 
product of this testing, the principal components solution for the December 
data was obtained. 

The December 1971 data consisted of 1089 observations provided 
by 109 students from three graduate curricula, 49 from Management 
(curriculum 817), 41 from Computer Systems Management, and 19 from 
Electrical Engineering. From a given list of instructors, each respondent 
ranked all he had known in the classroom environment on the 15 step 
ladder and then evaluated the top and Wotton five using the long form 
list of characteristics developed by Hildebrand and Wilson with addition 
of one more variable, (number 79). The applicability of each characteristic 


was indicated by (+) applied, (0) can't say or don't know, or (-) does 
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not apply. These responses were then converted into numbers and 
analyzed using BMDX72- Factor Analysis on the Naval Postgraduate 
School IBM-360/67 Computer. 

The results of the analysis on the first data set are shown in 
Table V. Each column contains ordered pairs, the first oí which is the 
original variable number from the long form list, followed by its factor 
loading. Where possible, the top ten loadings are shown in order of 
decreasing magnitude. It may occur that the magnitude of the ninth or 
tenth loading is not significant compared with the magnitude of the higher 
loadings, in this case only the significant loadings are included. In 
fact such a situation is desirable for it generally results in a very clear 
interpretation of that component. Such is the case for the component 
labeled Dynamism and Enthusiasm. At the top of each column is the 
component's interpretation, and at the bottom of each column, the asso- 
ciated eigenvalue and the cumulative percent of the total variance. 

The data set analyzed was heterogeneous in that respondents 
were from three different curricula. This was intentional because Hilde- 
brand and Wilson developed their methods using heterogeneous samples. 
It is observed that the five components found by them at the Davis 
Campus of the University of California also appear in this solution. 
However, the order of these components is changed, and there are two 
additional components, Evaluation Technique and Stimulation. Table VI 
lists ae components of this solution in order and also shows where 


they appeared in the Davis results. 
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TABLE VI 


COMPARISON OP PRINCIPAL COMPONENTS SOLUTION ORDER WITH 
HILDEBRAND AND WILSON SOLUTION 


DECEMBER 1971 COMPONENT H+W ORDER 
l. Organization and Clarity 2 
2. Instructor Individual Interaction 4 


3. Evaluation Technique - 
4. Synthetic/Analytic Approach 1 
5. Stimulation Е 
6. Dynamism and Enthusiasm 5 


7. Instructor Group Interaction 3 


39 





In order to investigate the effect of the curriculum as an exogenous 
variable, the individual curricula of the December 1971 data were sub- 
jected to principal components analysis separately. These solutions 
are shown in Tables VII, VIII, and IX. 

The last three solutions presented have differences in the order 
and number of the components resulting from the factor analysis. Most 
important is the difference in the first component of the Computer Systems 
Management Curriculum. Here Organization and Clarity appears as the 
last component while Dynamism and Enthusiasm is first. For the com- 
ponents associated with smaller eigenvalues, switching and loss or gain 
of the components can be expected from one solution to another. The 
first component makes the greatest contribution to the overall variance 
and a switch such as this indicates a significant difference due to the 
curriculum. It could be hypothesized that the shift took place because 
organization and clarity are common to the teachers in the computer 
discipline, and less variance in response to variables that constitute 
that component's interpretation was observed. Checking back to the 
means and variances of the original variables showed that instructors 
in this curriculum did score high on these variables, and the variances 
were smaller than those of variables that were associated with the other 
components. The evidence, at this point, indicated that it is possible 
that principal components solutions for this type of data may depend 
not леи on the make up of the student body, but also on the particular 


curriculum of the respondents. 
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The March 1972 data set, also collected under the sponsorship of 
the Faculty Council committee, provided an additional 278 observations 
by 47 students, 159 by 27 Operations Research/Systems Analysis students, 
49 by eight Oceanography students, and 70 by 12 Meteorology students. 
There was a difference in that each student evaluated the top, middle, 
and bottom two instructors from his ladder. The inclusion of the middle 
rankings was intended to provide insight into the effect on the principal 
components solution of a full range of teaching performance evaluations. 
This was felt to be appropriate since any methodology developed would 
be applied to all types of instructors, not just the best and worst. The 
first analysis was on the heterogeneous sample of all 278 observations. 
The results are shown in Table X. 

When compared to the December 1971-All Data solution, itis 
apparent that the components of Evaluation Technique and Dynamism and 
Enthusiasm do not appear. The first component is the same, however. 
It was not possible to say whether the inclusion of the middle ranked 
instructors caused a reduction in the variability of the variables which 
make up these components to such a degree that they failed to appear. 

en of the individual curricula of this data set was confined 
to the Operations Research/Systems Analysis Curriculum because the 
number of observations in each of the other two was too small compared 
to the number of original variables. This situation was found to result 
in meaningless solutions in this and prior projects employing factor 


analysis methods. The solution is shown in Table XI and,with the 
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exception of the loss of one component, is very much like the solution 
for all the March 1972 data. This might be expected since this curriculum 
provides 57 percent of the data. 

To provide a larger data base for heterogeneous solutions, the 
December 1971 and March 1972 data were combined. The 1367 observa- 
tions were subjected to principal component analysis, yielding the 
solution shown in Table XII. While this solution is not significantly 
different from the one obtained for the 1089 observations of the December 
1971 data alone, it is worth noting that four of the components had less 
than ten high factor loadings before a large magnitude drop in the loading 
was observed. This situation is desirable because it makes the interpre- 
tation of the components considerably easier. 

It was observed with this solution that the component previously 
interpreted to be Dynamism and Enthusiasm might be better labeled 
Presentation Technique. Reconsideration of the preceding solutions of 
this project and of the work of Hildebrand and Wilson supported this con- 
clusion. While they .chose the original interpretation, and that inter- 
pretation was ascribed to these solutions because of the recurrence of 
the contributing variables, not all of those variables can be considered 
to be contributing to Dynamism and Enthusiasm. All can, however, be 
ascribed to Presentation Technique. 

It was observed from the previous solutions that there appear to 


be seven components of effective teaching at the Naval Postgraduate 
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School, but not all show up in any given solution. It also appears there 
is a higher likelihood of all seven components resulting when using data 
from mixed curricula and with a large number of observations. 

The preceding work generates the following questions. Will a 
larger sample of data from a particular curriculum reinforce the results 
of a smaller sample from the same curriculum or tend toward the results 
of the larger mixed samples? What is the effect on the solution of the 
inclusion of the middle ranked instructors ? 

To answer the above questions, two additional sets of data were 
collected. The first was obtained in March 1973 and consisted of 243 
observations by 28 students from the Operations Research/Systems 
Analysis Curriculum on the three top, middle, and bottom ranked in- 
structors on their ladders. The second was obtained in May 1973 and 
consisted of 507 observations by 51 students from the Management 817. 
Curriculum on the five top and bottom instructors from their ladders. 
Ihe same 86 variable long form list of characteristics was used in both 
data collections. 

The principal components analysis solution for the March 1973 
data is shown in Table XIII. It differs from the preceding Operations 
Research/Systems Analysis solution in that Synthetic Analytic Approach 
does not appear while Evaluation Technique and Instructor Individual 
Interaction do. It cannot be said that this solution reinforces the 


preceding one for the same curriculum (see Table XI). 


49 








NATADIAYADO SISATUNVY SNOLLYAIdO 
-ININIVIAXA £461 HOYVN NOYI SNOILVAYISIO $76 UOI NOLILNTOS SININOINOD "WIIONITAd “IX Tavo 


ШЕКЕ 

























| | | 
| | 
| | | (35° | ev | 00%: саси Sine ` вопетед 
i | | О ел о 
| saryefnund 
| | 
| 0771 2671 го o 60:02 оптелпеба 
| | | рә1ето055ү 
| 
i 
| | Sz 786 | Uv Щи po" “08 01 
| | ЖА С 6Z°- ‘LL ov) CU 6 
| | | 9277 “14 Ip’ ‘Ss €£'- '49 Әр” “64 8 
| | | 62° “pS ot^ "ve | 56'- '22 US koy L 
| 65° ‘19 sv" “gg | 9e°- ‘OZ пети 9 
| | | cer “eL | 5-60 10-78 167 09 с 
| EB NU | 9p" 'vS | £v'- ‘08 O Р 
' | ES | ЖАНАТ) ЕО С € 
| es'- ‘TS соли с 2 
| 8S'- 798 вс" “ср I 
ні С züz МЕ = = = 2328 590 
СЕ HOG = ane a > 90 5б 2 
z $ dc = > 5 ЕЛ > = 
1 O 0 О ORO = = = 
Z > = = О 2 
Z, 
| 
i 


SNOILJINOSId ININOINOO 


50 





To investigate the eftect on the principal components solution of 
including the middle ranked instructors, it was necessary to have a data 
set with a sufficient number of observations such that when the middle 
ranked instructors were removed, the remaining number of observations 
was large enough with respect to the number of original variables. Neither 
of the data sets that had evaluated such instructors was large enough, so 
the two were combined. They were then analyzed with and without the 
observations on the middle ranked instructors. The solution for the 
combined set of 402 observations had six components, all of which had 
appeared in either of the two solutions of the March data. With the 
exception of some minor shifting of the order of components with small 
eigenvalues, the solution for the set with the middle ranked instructors 
removed was the same. Inclusion of these observations had no effect 
on the principal components solution. 

The results of the principal components analysis of the May 1973 
data are shown in Table XIV. The results for this data set were somewhat 
startling. All previous solutions, with the exception of one, had 
Organization and Clarity as the first component. Here the component 
split with Clarity coming first with the greatest amount of the total 
variance, followed by Organization. As might be expected, when the 
covariation of those variables associated with the component interpreted 
as Clarity with those associated with the component interpreted as 
Organization was checked, the values were comparatively low. It was 


apparent that the second group of Management 817 students did not 
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feel the same way about the instructors that the first group had. This 

fact is also illustrated by the merging of the two components previously 
defined as Instructor Individual Interaction and Instructor Group Interaction 
into one component called Instructor Student Interaction. 

There were nine instructors that were evaluated by both of these 
groups. They were ranked according to their average range scaled ladder 
scores for both groups and the Spearman Rank Order Correlation Coef- 
ficient was computed and found to be .27. This is further evidence that 
any two groups of respondents from the same curriculum may look at 
instructors in a different manner. 

Table XV shows the principal components solution for all Manage- 
ment 817 data collected and Table XVI shows the same for all data 
collected for this research project. 

It appears that even though the combined Management 817 solution 
reinforces the December 1971 Management 817 solution, it is also tending 
toward the solutions for the mixed December 1971 All and March 1972 АП 
solution, as well as the solution for all collected data. It also is 
evident that there are fewer original variables with high enough loadings 
to be used in component interpretation as the number of observations 
gets higher. This is clearly the case for the 2107 observations from all 
data sets where at least ten significant loadings could not be found for 


five of the seven components. 
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Ва RANKING 

The use of instructors average range scaled scores computed from 
the student ladder rankings has been mentioned in conjunction with showing 
that the overlapping instructors of the Management 817 curriculum data 
sets were not perceived, by the two groups of students, in quite the 
same manner. The one dimensional ranking based on these scores is the 
simplest representation of how a group of respondents ranks the individual 
instructor on the average. At the same time the May 1973 Management 
817 data was being collected from 51 students of that curriculum, а 
short form developed from the preceding Management 817 data's principal 
components solution was administered to 29 students of the same input. 
Both groups ranked the same set of instructors on the same one dimen- 
sional ladder. The two groups' average rankings based on the average 
range scaled scores were compared and found to have a Spearman's Rank 
Order Correlation Coefficient of .958. The short form group's ranking 
compared with the December 1971 group's for the nine overlapping 
instructors resulted in a coefficient of .35. Here is additional evidence 
that these two groups from the same curriculum and same input, did not 
perceive the instructors as did the earlier group. 

All data sets mentioned in the preceding section were factor 
scored once the principal components solutions had been found. It was 
hoped that these scores could be used to find a means of converting 
the reduced data space model of the students' perceptions to one dimension. 


It was observed that assigning numerical values to the long form 
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variables, such as (+1, 0, -1) for (applicability, don't know, non- 
applicability), adding the 86 responses and ranking the evaluated in- 
structors on the resulting sums, resulted in a fairly accurate reproduction 
of a student's ladder. It was decided to use the factor scores in a 
Similar manner. First, the average factor score vector was computed 

for each instructor. Then four rankings were constructed based on the 
sum of the averaged factor scores, the sum of these scores weighted by 
‘the eigenvalue, the sum of these scores weighted by the square root of 
the eigenvalues, and the sum of these scores weighted by the cube root 
of the eigenvalues. The weighting schemes were an E to use the 
relative importance of the individual components on which each factor 
score was computed. The Spearman Rank Order Correlation Coefficients 
for these rankings compared to that based on the averaged range scaled 
scores for the December 1971 Management 817 data set were .65, .80, 
.786, and .753 respectively. The implication from these results appears 
to be that the variability of each component, as manifested by the asso- 
ciated eigenvalue, should be considered in such ranking schemes if 
reproduction of the one dimensional ranking is desired. It is not immed- 
iately clear which of the above schemes is best since the three weighted 
ones are fairly close in the rank order correlation coefficients and their 


difference is not statistically significant. 
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С^. MENTIRAS NS E TEACHING PATTERNS FROM FACTOR SCORES 

Having identified the components of effective teaching by principal 
components factor analysis of the varicus data sets, effort was turned to 
attempting to determine patterns or styles of teaching based on these 
components. Such patterns may be helpful to the instructor seeking to 
improve himself and to the evaluating administrator seeking to classify 
instructors on teaching performance. 

While it is theoretically possible to identify teaching patterns 
based on the data from the long form questionnaires, it is not practical 
for two reasons. First, computing requirements for that many variables 
taxes the memory assets of the machine. Second, explaining such 
patterns would be extremely difficult. The parsimmonious representation 
of the original data by the factor scores which retain 45-50 percent of 
the information while reducing the number of variables by 90-95 percent 
provides a tenable solution for both of the problems. 

To identify the teaching patterns, the average factor scores of 
each instructor evaluated in a particular data set would be cluster 
analyzed and the means of each cluster would be hypothesized as de- 
fining a particular pattern or style of teaching. The immediate problem 
is that cluster solutions of any size,up to the extreme of each vector of 
scores being a single cluster,are possible. It was thought that ranking 
reproduction or discriminant analysis or both could be used to find the 
solution for which the homogeneous groups of instructors defined by the 


clusters would be most distinct from each other. 
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To test this approach, the December 1971 Management 817 data 
set factor scores were averaged for each instructor and the averaged 
scores cluster analyzed by a k-iterative means clustering program. S 
Solutions for two through seven clusters with the associated vector of 
means for each cluster in the solution are shown in Table XVII. Instructors 
are represented by letters of the alphabet to maintain confidentiality, 
the letters being assigned according to the average range scaled score 
ranking based on the student ladders. This ranking appears in the left 
hand column. The individual clusters are arranged to preserve this 
ranking within the cluster to provide insight as to the ranking reproduction 
ability of each solution. Note that, by mere inspection, it appears that 
the four cluster solution best reproduces the alphabet. Solution five 
and six appear almost as good as each other. 

Because the number of groups that can be discriminant analyzed 
must be less than or equal to the number of variables, work in this area 
was limited to seven clusters. Cluster solutions two through seven were 
subjected to a discriminant analysis program. » For this method of 


analysis, a test statistic is computed based on the null hypothesis that 


TEE D. J., MICKA: A Fortran IV Iterative K- means Cluster 


Analysis Program, CTB/McGraw Hill: Del Monte Research Park, 
Monterey, California, September, 1970. 


16 | еке | 
The discriminant analysis program consisted primarily of the 


subroutines DMATX and DISCR from the IBM Scientific Subroutine Package 
available at the Naval Postgraduate School Computer Center. 
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TABLE XVII 
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the means of the clusters are the same. A test statistic greater than 
the acceptance value implies that there is sufficient discrimination 
between the means cf the clusters to reject the null hypothesis. The 
greater the magnitude, the greater the discrimination. 

The test statistic is the Mahalanobis D which can be 
considered to be a Chi-square variate with m(g-1) degrees of freedom, 
m being the number of variables, g the number of groups or clusters. 
Figure 3 shows the plot of the test statistic for the associated cluster 
solution that was put through the discriminant analysis program. The 
greatest value of the statistic occurs for the six cluster solution. Re- 
ferring back to Table XVII there is an intuitive appeal associated with 
this solution for it represents the top and bottom few instructors of the 
Original rank structure with high and low means respectively. These 
are the types of teaching patterns that might be expected for these 
extremes in the spectrum of teaching performance. Nothing, however, 
can be said regarding the middle of any of the cluster solutions being 
labeled good or bad by virtue of the cluster means. 

The rank order ne for each of the cluster solutions in 


Table XVII with the average range scaled score ranking are as follows: 


CLUSTER SOLUTION COEFFICIENTI 
TWO 2090 
IHREE 2610 
FOUR .944 
FIVE 1966 
SIX 1933 
SEVEN SS DO 
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TEST STATISTIC AND CHI-SQUARE ACCEPTANCE VALUE 


PENE ES. PLOr OF Test STATISTIC AND 
Car OUA L ACCEPTANCE VALUE VERSUS THE 
NUMBER OF CLUSTERS FOR THE INSTRUCTOR 

AVERAGED FACTOR SCORES FROM DECEMBER 1971 
MANAGEMENT 817 DATA 


O- TEST STATISTIC 


1500 
+- ACCEPTANCE VALUE 


1200 


900 
© 
© 
600 | 
300 
© 
е 
+ + 
5: 
2 3 4 5 


CLUSTER SOLUTION NUMBER 
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It appears from these results that solutions for four, five, and six 
clusters are about equal in rank reproduction. 

To e comparison of the individual groups within the cluster 
solutions with high rank order correlations, a graphical representation 
of group means for each component of effective teaching is contained 
in Appendix B. Cluster solutions 4, 5, and 6 shown in Table XVII, have 
the individual clusters numbered to provide a cross reference between 


the table and the appendix. 


D. SHORT FORM AND TEACHING PATTERNS BASED ON IT 
Unfortunately, the use of factor scores to determine teaching 
pattern or styles has only experimental merit. 1t would hardly be practical 

to administer a long form questionnaire each time it was desired to 
evaluate a group of instructors, subject the data to principal components 
analysis, factor score it, then go through cluster and discriminant | 
analysis. The determination would be much easier if a short form of 

the few components of effective teaching, known to be perceived by the 
respondents, as ne could be used to acquire the same information. 
Such a form was designed and used successfully by Hildebrand and 
Wilson. 

An initial problem with the above approach was that for the initial 
Naval Postgraduate School data analyzed by the principal components 
method, no clear general solution had been found. Because there might 
be differences between curricula, it was decided to restrict this approach 


to a single curriculum, in this case the Management 817 Curriculum. 
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This curriculum was chosen because it was the largest of the individual 
data sets and it was felt that it would be possible to administer separate 
long and short forms to two sufficiently large groups simultaneously 
because of the usually large inputs to the curriculum. Correlation 
studies could then be done between the long and short form results. 
Table XVIII shows the short form variables that were developed from 
the principal components analysis of the December 1971 Management 817 
data. It was administered in May 1973 to 29 students of the Management 
817 curriculum while 51 students of the same curriculum were administered 
the long form. As may be recalled from section A. of this chapter, the 
May 1973 principal components solution was not at all similar to the 
December 1971 solution. So while responses to the short form were 
obtained, it was not possible to compare them with the long form results. 
It was decided to apply the previously described methods of cluster 
and discriminant analysis to this data to see if teaching patterns could 
be obtained, even though it was not clear whether the short form 
variables reflected the respondents perception of the components of 
effective teaching. These results are shown in Table XIX and Figure 4. 
Here again, the six cluster solution was indicated as best by discriminant 
analysis. Note that the means are now all positive because the responses 
were made ona scale of 1-7. The Spearman Rank Order Correlation 
Coefficients of the cluster solutions with the average range scaled 


score ranking were: 
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CLUSTER SOLUTION COEFFICIENT 


TWO ee 
THREE . 870 
FOUR IS Y 
FIVE .947 
SIX 556 
SEVEN - .964 


Graphical representation of cluster solutions 4, 5, and 6 is provided 
in Appendix C in the same manner as was done for cluster solutions on · 


factor scores in section C. 
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TABLE XVIII 


SHORT FORM VARIABLES DEVELOPED FROM DECEMBER 1971 MANAGEMENT 
817 CURRICULUM 


ie: He makes himself clear, makes his objectives known, summarizes 
major points, is well prepared, presents material in an organized manner, 
is aware of class progress and understanding. 


2. He is sensitive to the response of the class, encourages student 
participation through questions and discussion and is willing to deviate 
from his prepared lecture or course outline to pursue points resulting 
írom student contributions. 


B. He is interested in students as individuals, treats them with respect 
and as equals, is available for individual consultation outside of class, 
and is friendly and courteous to students. 


4. He has a command of his subject, discusses recent developments 
in the field as well as contrasting theories or points of view, has an 
analytical method of presenting material that includes the origins of ideas 
and concepts as well as full development of the subject. 


S. He gives exams which require creative and original thinking, nave 
instructional value, stress conceptual understanding, require synthesis 
of various parts of the course, and permit students to show their under- 
standing of the subject. 


D: He is a dynamic and energetic person who is enthusiastic about 
both his profession and the subject he is teaching. He has an interesting 
style of presentation and is an excellent public speaker. 


Ж His lectures and assignments have stimulated you such that you 
have an increased appreciation of the subject, have learned new ways to 
evaluate problems, and have studied a topic from the course on your 
own initiative. 
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TABLE XIX 


CLUSTER SOLUTION 
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TEST STATISTIC AND CHI-SQUARE ACCEPTANCE VALUE 


Pieri, LOT OF TEol STATISTIC AND CHI-SQUARE 
ACCEPTANCE VALUES VERSUS THE NUMBER OF CLUSTERS 
FOR INSTRUCTOR AVERAGED SHORT FORM SCORES FROM 

THE MAY 1973 MANAGEMENT 817 DATA 
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IV, CONCLUSIONS 

A. Pity wa hiON Or COMPONENTS OF EFPECTIVE TEACHING AT 

THE NAVAL POSTGRADUATE SCHOOL 

The primary objective of this research was to use the methods of 
Hildebrand and Wilson to determine the components of effective teaching 
at the Naval Postgraduate School. What has been determined is that 
their methods will find the components of effective teaching for any 
group of respondents, but the solutions for one group are different from 
another in the number of components found, the order of the components, 
and the original variables that are used to interpret the components. 
Such differences occur even when the groups are from the same curriculum. 
On the other hand, there seems to be a set of components which exist to 
varying degrees for all groups analyzed. This set is most clearly defined 
in solutions for large heterogeneous groups of respondents. The seven 
components which define this set listed in their order of contribution to 
the overall variability of the Principal Components solution on all the 
data collected in the course of this research are: 

Е. | ORGANIZATION AND CLARITY 

2% INSTRUCTOR INDIVIDUAL INTERACTION 

ци INSTRUCTOR GROUP INTERACTION 

4. SYNTHETIC ANALYTIC APPROACH 

S EVALUATION TECHNIQUE 

б. PRESENTATION TECHNIQUE 


7. STIMULATION 
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TABLE XX 


SUMMARY OF ALL PRINCIPAL COMPONENTS ANALYSIS SOLUTIONS 


ORGANIZATION 
AND CLARITY 
INSTRUCTOR 
INDIVIDUAL 


WESEMBER 1971 
ALL CURRICULA 
1089 OBSERVATIONS 


DECEMBER 1971 
MANAGEMENT 817 1 
498 OBSERVATIONS 


DECEMBER 1971 
COMPUTER SCIENCE MGT 6 
402 OBSERVATIONS 


DECEMBER 1971 
ELECTRICAL ENGINEERING 1 
189 OBSERVATIONS 


e 


MARCH 1972 

ALL CURRICULA 1 
278 OBSERVATIONS 

MARCH 1972 

OPERATIONS ANALYSIS 1 


159 CBSERVATIONS 


DEC 71 + MAR 72 
ALL CURRICULA | 1 
1368 OBSERVATIONS 


MARCH 1973 


OPERATIONS ANALYSIS 1 
243 OBSERVATIONS 

MAY 1973 

MANAGEMENT 817 2/1 


507 OBSERVATIONS 


DEC 71 + MAY 73 
MANAGEMENT 817 l 
995 OBSERVATIONS 


ALL DATA 
2107 OBSERVATIONS 


HILDEBRAND AND 
WILSON SOLUTION 2 
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Table XX shows a summary of all principal components solutions 
on various data sets and subsets collected in conjunction with this 
research. The last solution shown is that of Hildebrand and Wilson in 
their work at the University of California, Davis Campus. It is presented 


in support of the conclusions in the preceding paragraph. 


Б. DEVELOPMENT OF A SHORT FORM QUESTIONNAIRE 

In addition to the components of effective teaching, it was hoped 
that a short form questionnaire could be developed, using such components, 
that would extract an accurate representation of the students' opinion of 
teaching performance. The smaller number of variables would be easier 
to analyze and would provide useful, composite information to both the 
instructor and the evaluating administrator. It was shown that while 
the methods of Hildebrand and Wilson in this respect can be emulated, 
the short form developed from a parent Principal Components solution did 
not have a one to one correspondence of variables to the Principal Com- 
ponents solution for a counter part group of long form respondents. As 
a result, the short form was not validated and no conclusion can be made. 
It is felt that the approach applied to larger heterogeneous samples both 
for development and validation would work as it did for Hildebrand and 


Wilson. 


OF IDENTIFICATION OF TEACHING PATTERNS 
It is concluded that teaching patterns or styles can be found using 


cluster analysis to group instructors on variables which are the components 
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of effective teaching as perceived by the student and then using dis- 
criminant analysis to find the cluster solution which is best. This 
method can be used for either factor scores or the short form variables 
developed from the Principal Components solution. On the other hand, 
it is not possible to make a judgement about performance of any cluster 
other than the extremes which have universally high or low means and 


are clearly the good and bad performers, respectively. 


Да 





V. COMPARISON OF RESULTS WITH OTHER WORK 


At the same time this research was being carried out, another 
approach was being used to determine what characterized effective 
teaching at the Naval Postgraduate School. y This approach employed 
the method of critical incidents coupled with content analysis of state- 
ments by the respondents. There were two statements from each re- 
spondent, one describing a time when he was particularly satisfied with 
an instructor, the other describing when he was not. A decision tree was 
developed to classify key phrases and an order-of-importance listing of 


categories was developed. That listing follows: 


Ir Course Organization 
2. Evaluation of Student 
3. Attitude Toward Student Understanding 
4. Ability to Create Learning Environment 
3: Instructor Preparation 
6. Ability to Explain 
И Instructor Knowledge 
8. Ability to Teach at the Appropriate Level 
ЭС Instructor Attitude Toward Course 
10. Lecture Organization 
P. Attitude Toward Student 
12, Attitude Toward Questions 
S Student Learning Result 
14. Instructor Control 
hon Instructor Availability 
16. Ability to Evoke Interest 
17 


БШЕ Шапса enson, J]. N., An Analysis of Student 
Perceptions Concerning Instructor Effectiveness at the Naval Postgraduate 


School, Master of Science Thesis, Naval Postgraduate School, Monterey, 
@Walitornia, 1973. 
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It is possible, by studying the comments and phrases that went into the 
development of this listing compared with variables that played the most 
important role in the development of the components of effective teaching 
from the principal components analysis method, to identify these cate- 


gories with the components. Using the numbers that appear above: 


Organization and Clarity Ток О ЮГО 
Instructor Individual Interaction 115715 
Instructor Group Interaction swe la el 
Synthetic Analytic Approach 7, 4 
Evaluation Technique 2 
Presentation Technique (D+E) 9 
Stimulation №5, 16 


Such identification is not completely accurate for the methods are in- 
herently different and one tends to lose some of what the other picks up. 
This is understandable when one recognizes that the principal components 
method uses the variance structure of the data, while the tabulation of 
critical incidents is concerned primarily with frequency. Hence a teacher 
characteristic held universally salient would be picked up by the latter, 
but, because of the small variability, would be lost by the former. It 
cannot be argued that one method is better than the other, but rather 


they should be considered as mutually supporting. 
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APPENDIX A 


SUMMARY OF ALL DATA COLLECTED 


i LONG FORM DATA 
NO. NO. INSTRUCTORS 
DATE CURRICULUM STUDENTS OBSERVATIONS EVALUATED 
MANAGEMENT 49 498 TOP AND 
ШЕТ COMPUTER BOTTOM 
SYSTEMS 41 402 FIVE 
MANAGEMENT 
ELECTRICAL 
ENGINEERING 19 189 
OPERATIONS TOA 
RESEARCH 2 159 MIDDLE, 
AND 
2 
MAR 7 OCEANOGRAPHY 8 49 HOC 
METEOROLOGY 12 70 TWO 
MAR 73 | OPERATIONS 28 243 TOP, 
RESEARCH MIDDLE, 
AND 
BOTTOM 
THREE 
MAY 73 MANAGEMENT 51 507 TOP AND 
BOTTOM 
FIVE 
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il. SHORT FORM DATA 


NO. NO. 
DATE CURRICULUM QTR. STUDENTS | OBSERVATIONS 
OPERATIONS 
RESEARCH 8 31 SEP 
6 51 612 
4 63 756 
8 7 84 
MAR 72* OCEANOGRAPHY 6 14 168 
4 16 192 
8 16 187 
METEOROLOGY 6 7 TE 
4 12 151 
OPERATIONS 8 23 253 
MAR 73** RESEARCH 
4 33 363 


MAY 73 MANAGEMENT 


* The March 1972 short form data was based on the use of the 
short form variables developed by Hildebrand and Wilson. When it was 
discovered that there were possible additional characteristics of effective 
teaching perceived by students at the Naval Postgraduate School and that 
the order of those characteristics that were similar to theirs was not the 
same, efforts were devoted to development and validation of a short form 
based on the Naval Postgraduate School characteristics. As a result, 
the March 1972 short form data was not analyzed. 

** The March 1973 short form data was based on an erroneous set 


of characteristics of effective teaching and was not analyzed. 
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APPENDIX B 
GRAPHICAL REPRESENTATION OF INDIVIDUAL CLUSTER MEANS FOR THE 
БЕЛЕ FIVE, AND SIX#-LUSTER SOLUTIONS ON THE AVERAGED FACTOR 
N@Oneo hon DHE INSTRUCTORS EVALUATED IN DECEMBER 1971 BY THE 


MANAGEMENT 817 CURRICULUM 
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APPENDIX C 
GRAPHICAL REPRESENTATION OF THE INDIVIDUAL CLUSTER MEANS FOR 
THE FOUR, FIVE, AND SIX CLUSTER SOLUTIONS ON THE AVERAGED SHORT 
FORM SCORES FOR THE INSTRUCTORS EVALUATED IN MAY 1973 BY THE 


MANAGEMENT 817 CURRICULUM 
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